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abstract 



Although most school districts and schools are 



routinely involved in data collection, they do not typically use the 
data they collect in a systematic fashion to identify strengths and 
weaknesses and develop improvement strategies. One reason for the 
lack of data use is the perception that the data are being collected 
for someone else's purposes. Without taking steps to gather 
systematic, representative information, data collected in this way 
may lead to inappropriate cone 1 us i ons and actions. State^provided 
data often do not 1 ead to local improvement for a variety of similar 
reasons. Program evaluations are another source of information on 
local educational performance, but expert research is often ignored 
or devalued. Locally developed performance indicator systems offer a 
promis ing strategy for establishing a data^based program improvement 
process in districts and schools. By encouraging local educators to 
articulate their goals and involving them in deciding how to measure 
their performance on the goals, one model, "At Your Fingertips," 
ensures that indicator systems will be relevant to local educational 
objectives. This program improvement process follows six steps: (1) 
identify goals; (2) identify related outcomes, practices, and inputs; 
(3) identify data sources; (4) development indicators for outcomes, 
practices, and inputs; (5) examine and interpret data; and (6) 
develop improvement strategies. (A selected bibliography lists 51 
items.) (YLB) 
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Q Most school districts and schools in 
this country are routinely involved in 
data collection. Administrators tally av- 
erage daily attendance (ADA) rates and 
maintain transcript data, including stu- 
dents' course enrollments and grades. As 
a condition of receiving state or federal 
funds, they collect information on par- 
ticipants in particular programs or 
activities. Administrators also rely on an- 
ecdotal information to assess informally 
the quality of teaching and learning at 
their site, and teachers and counselors 
use various cissessment instruments for 
diagnosing individual students. Thus, 
school districts and schools collect a 
wide array of data. However, they do 
not typically use the data they collect in 
a systematic fashion to identify strengths 
and weaknesses at their sites and to de- 
velop improvement strategies. 

One reason for the lack of data use is 
the perception that the data are being 
collected for someone else's purposes. 
Administrators report ADA to the state 
education agency, provide transcripts to 
postsecondary institutions, and report 
grades to students and parents. Similarly, 
participant information is reported to the 
state or federal office that funds a par- 
ticular program, and test scores are 
maintained by the teacher or counselor 
who administers the test. Rather than 
conoidering these various data as poten 
tial sources of information on the quality 
of leaching and learning at a site, educa- 
tors view them as obligatory or 
otherwise limited in value. When '‘data” 
are used, they often take the form of an- 
ecdotes or casual observations. Without 
taking steps to gather systematic, repre- 
sentative information, data collected in 
this way may lead to inappropriate con- 
clusions and actions. 

State agencies sometimes provide 
data to school districts and schools that 



are intended to trig '^er school improve- 
ment efforts. The 'hool report card 
movement that b( i during the 1980s 
and Perkins-mauaated performance 
mecisures and standards for vocational 
education, are included in this category. 
For example, state education agencies in 
California and Illinois produce annual 
report cards summarizing various 
performance data-in particular, state 
achievement test scores-for each school 
district in the state. In response to the 
1990 Perkins Act requirement, some 
states distribute performance reports to 
their local vocational program adminis- 
trators. The reports include such 
information as achievement test scores 
and rates of placement into employment 
and further education for vocational stu- 
dents or completers. These state-level 
efforts to provide data to local educators 
have produced mixed results. 

State-provided data often do not lead 
to local improvement efforts for a variety 
of reasons. The school report card expe- 
rience shows that superintendents and 
principals may find district- and school- 
level data useful for public relations 
purposes. However, teachers typically do 
not find these data to be useful for as- 
sessing their own performance or the 
performance of their students. In other 
cases, local administrators and educators 
find that the state provided data do not 
reflect what they are trying to do in their 
community. The joint RAND-Manage- 
ment Planning Research Associates 
(MPR) study for the National Center for 
Research in Vocational Education 
(NCRVE) of the preliminary effects of 
Perkins performance measures found 
that local voca tional administrators and 
instructors were more likely to use the 
data if they had personally participated 
in developing the performance measures 
and related assessment instruments 
(Stecher et al., 1995). Allowing for a cer- 
tain degree of local customization of 
perform- ance measurement data im- 
proves the chances that local educators 
will find the data to be meaningful and 
relevant. When state-provided data do 



not reflect local educational goals or do 
not describe a useful unit of analysis, the 
data may simply be ignored. 

Program evaluations are another 
source of information on local educa- 
tional performance. When a new pro- 
gram is implemented, efforts may be 
made to evaluate its effectiveness. Typi- 
cally, districts and schools rely on 
outside evaluator? to undertake this 
work, although in some larger school 
districts, the district staff may include an 
evaluator. Responsibility for the evalu- 
ation is usually given to an expert, 
because a rigorous evaluation-particu- 
larly one that is intended to produce an 
estimate of program impact-requires at- 
tention to exacting methodologies. 

However, expert research is often ig- 
nored or devalued for many of the 
reasons described above. Administrators 
and faculty may perceive the evaluation 
as providing someone else with informa- 
tion about their program; may question 
the focus and goals of the evaluation; 
and may react defensively lo seemingly 
critical results, by dismissing them or ex- 
plaining them away. Consequently, 
evaluation findings are often “underutil- 
ized.” In an effort to improve using 
evaluation results, some educators have 
encouraged building the capacity of dis- 
tricts and schools for self-evaluation. 
However, implementing a rigorous 
evaluation design (involving random as- 
signment of subjects to treatment and 
control groups, identification of an ap- 
propriate comparison group, or sta- 
tistical equation of participant and non- 
participant groups) often proves too 
burdensome or is practically infeasible 
for administrative or political reasons. 
Moreover, evaluation tends to be a one 
time activity, which does not encourage 
ongoing improvement efforts. 

Experience on NCRVE projects and 
a review of the evaluation utilization and 
performance indicator literatures suggest 
several strategies for improving the like- 
lihood that performance data will be 
used-and used well-by local educators.^ 
These include moving from a framework 



^ * These NCRVE projects include providing lechnicai assistance to the Southern Regional Education Board Stale Vocational 

^ Education Consortium, working with New Castle County Vjcalional-Technical High School District in Delaware lo develcm a school improve- 
ment process, and providing technical assistance to the sUtes on implementing Perkins performance measures and standards. 
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where data are reported to someone else 
toward a framework where data are used 
locally; involving local educators in de- 
signing performance measurement sys- 
tems; and providing technical assistance 
to increase the capacity of local educa- 
tors to use data critically. 

At Your Fingertips 

MPR Associates' staff are midway 
through a two-year NCRVE project to 
develop training materials that provide 
step-by-step guidelines for setting up per- 
formance indicator systems. At Your 
Fingertips: Using Data for Program Improve- 
ment will produce a workbook and 
trainer’s manual that introduce educators 
to a practical method for using locally 
available data to determine strengths 
and weaknesses, identify improvement 
strategies, and monitor progress. The 
materials are based on experience 
gained from working inten- sively with a 
number of sites as well as relevant re- 
search.^ 

What Are Performance Indicators 
and Systems? 

Performance indicators are statistics 
that “indicated” something about the 
performance or health of a district, 
school, or program. Indicators describe 
crucial educational outcomes, processes, 
and inputs, and typically appear as aver- 
ages, percents, or rates. A performance 
indicator systern establishes loose rela- 
tionships among the outcome, process, 
and input statistics, and enables educa- 
tors to monitor these statistics on an 
ongoing basis. Such a system helps to 
identify strengths and weaknesses and 
generates discussion about causes and 
appropriate improvement strategies. Ul- 
timately, a performance indicator system 
produces evidence about whether strate- 
gies are working or not. 

Comparing Performance Indicator 
Systems and Evaluations 

At Your Fingertips focuses on estab- 
lishing performance indicator systems 



rather than on implementing a formal 
evaluation design. While formal evalu- 
ations may represent a one-time or 
periodic activity, performance indicator 
systems are designed to support continu- 
ous program improvement. Moreover, 
formal evaluations often prove overly 
burdensome or impractical to imple- 
ment, whereas all districts and schools 
should have ready access to at least 
some relevant indicator data. A primary 
objective of the project is to encourage 
local educators to become familiar and 
comfortable with using data for program 
improvement. Performaiice indicator 
systems offer a more appropriate strat- 
egy' than formal evaluations to achieve 
this end. 

Performance indicator systems differ 
from formal evaluations in several ways 
that are listed in Table 1. Performance 
indicator systems are primarily descrip- 
tive, while formal evaluations provide 
causal evidence about the impact of par- 
ticular strategies or activities. Indicator 
systems help answer the question, “How 
well do our collective strategies appear 
to be w'orking?” In contrast, evaluations 
help answer the question, “What is the 
unique contribution of a particular strat- 
egy or activity?” 

However, the two approaches can be 
complementary. By providing informa- 
tion on crucial aspects of schooling, 
performance indicator systems may help 



identify areas that require more thor- 
ough evaluation. For their part, formal 
evaluations may help identify serious 
conditions that should be monitored 
on an ongoing basis through indicator 
systems. 

The Program Improvement Process 

At Y)ur Fingertips describes a six-step 
program improvement process that is 
illustrated in Figure 1. 

A Goal Driven Process 

Performance indicators should be 
rooted in local goals. If not, they may 
end up becoming the de facto Edu- 
cators are encouraged to identify what it 
is they are striving to achieve in their dis- 
trict, school, or program, and then what 
information they need to determine 
whether they are achieving these goals. 
A v»^ide variety of education stakeholders 
should be involved in the process of 
identifying goals. These stakeholders 
may include ac .aemic and vocational 
teachers, counselors, school- and dis- 
trict-level administrators, school board 
members, state education agency staff, 
parents and students, local employers, 
and local postsecondary institutions. 
Generally, all those who have a stake in 
educational outcomes or who will be re- 
sponsible for helping to achieve the 



Table 1 

INDICATOR SYSTEMS COMPLEM ENT EVALUATIONS 
PerfoiTnance Indicators I Formal Evaluations 



• Ongoing 

• Describe the district, school, or 
program 

• Indicate progress and achievements 

• Suggest areas for improvement 

• Monitor changes over time 



I • One-time or periodic 

' • Formally evaluate a program 

[ • Isolate the impact of particular 
I activities 

• May provide valid comparisons of 
participants and nonparticipants 



^ Sites included a vocational- technical high sc!)ool district, a joint academic and vocational hiah school, an a statewide youth corrections 
education program. In addibon to these sites, many other high schools and community colleges as well as state-level administrators participated 
in introductorv workshops on the At Your Fingertips materials, 
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Figure 1 

THE PROGRAM IMPROVEMENT PROCESS 
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goals should participate in establishing 
them. 

Outcomesj Practices, and Inputs 

Performance indicator systems may 
be b^lsed on a simple model of the 
schooling process that incorporates three 
basic elements: (1) student outcomes, (2) 
school practices (or processes), and (3) 
school inputs. Although some perform- 
ance measurement initiatives have 
focused on just one of these elements to 
the exclusion of the others, performance 
indicator systems provide a more power- 
ful analytic tool when they collect 
information on all three. 

Student outcomes describe the ulti- 
mate end product of the education 
system-or w'hat we want students to 
know or achieve. Examples of student 
outcomes include academic achieve- 
ment, employability or work-readiness 
skills, high school graduation, and place- 
ment into and success in further 
education or employment, among many 
others. School practices contribute to 
student outcomes. Examples include the 







curriculum, instructional strategies, and 
supporting structures such as scheduling 
practices. School inputs describe the 
background for both practices and out- 
comes. They are typically considered to 
be “givens;" that is, they represent con- 
ditions that are difficult to change, such 
as student demographics, local economic 
conditions, facilities, and school funds. 
Working from the goals id tified in 
Step 1, educators are encouraged to 
identify crucial outcomes, practices, and 
inputs and their relationships to one an- 
other. In effect, educators develop 
explicit hypotheses about the schooling 
process at their site. Then the resulting 
performance indicator data will allow 
them to test their hypotheses. 

IdentiJyiTig Data Sources and Develop- 
ing Indicators 

The next steps in the process of devel- 
oping a performance indicator system 
involve identifying data sources and de- 
veloping indicators that describe the 
outcomes, practices, and inputs identi- 
fied in Step 2. Educators should begin by 



identifying data sources that are already 
maintained by districts or schools. Only 
when existing data sources do not pro- 
vide sufficient information do par- 
ticipants need to consider special data 
collection efforts. Examples of com- 
monly available data sources include the 
follov/ing: 

■ enrollment records (enrollments, 
transfers, and dropouts) 

■ average daily attendance records (for 
students, teachers, and administrators) 

■ transcripts (course enrollments, cred- 
its earned, course levels, and grades) 

■ student records (student demograph- 
ics, and extracurricular activities) 

■ standardized achievement test scores 

■ guidance records (development of ca- 
reer plans, and participation in 
guidance activities) 

■ disciplinary action records (referrals, 
infractions, detentions, and suspen- 
sions) 

■ student follow up surveys (employ- 
ment and wage rates and further 
education) 

■ college entrance exams (scores and the 
proportion of students taking the test) 

■ funds and expenditures (per pupil ex- 
penditure trends) 

■ community surveys or needs assess- 
ments 

Examples of special data collection ef- 
forts that may be developed to 
supplement existing data sources include 
special surv'eys and questionnaires, inter- 
views and focus groups, teacher logs and 
diaries, classroom observations, and al- 
ternative assessment instruments. Edu- 
cators should identify those new data 
sources that are most essential to de- 
scribing identified outcomes, practices, 
and inputs, and should plan to phase 
these into their system. 

Once data sources have been identi- 
fied, educators are ready to develop 
actual indicators. As mentioned pre- 
viously, indicators are statistics that 
typically appear as averages, percents, 
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and rates. Examples of performance in- 
dicators include average achievement 
test scores and high school graduation 
rates. However, it is generally a sound 
practice to select multiple indicators for 
each outcome, practice, or input. Teach- 
ing and learning are complex processes, 
and a single indicator will rarely ade- 
quately describe a particular construct or 
concern. For instance, if a school’s goal 
is high academic achievement for all stu- 
dents, then educators may want to know 
what percentage of graduates complete 
high-level academic coursework, what 
proportion of teachers report integrating 
academic and vocational learning on a 
regular basis, as well as what the average 
achievement test score is for the school 
and how it is increasing or decreasing 
over time. Collecting data on one of 
these indicators to the exclusion of the 
others may miss important information 
on academic achievement and distort 
perceived performance. 

Interpreting the Data and Developing 
Improvement Strategies 

The final steps of the program im- 
provement model involve interpreting 
indicator data and developing improve- 
ment strategies. Sound interpretation 
requires determining the appropriate stu- 
dent cohort to examine on each 
indicator (for examole, last year’s gradu- 
ates, this year’s seniors, or this year’s 
ninth graders); determining the appro- 
priate unit of analysis (such as the 
school, a grade level, or individual class- 
rooms); and identifying important 
subpopulations (for instance, examining 
data by gender and race-ethnicity). 
Through the At Your Fingertips materi- 
als, participants in the program 
improvement process also become fa- 
miliar with some basic statistical 
concepts to assist them in analyzing the 
data. After discussing what the data 
mean, participants develop appropriate 
improvement strategies. To do so, educa- 
tors must decide when they believe they 
have sufficient information to proceed 
with specific strategies and when they 
need more or different data. 




Dynamic Process 

Developing an indicator system for 
program improvement purposes re- 
quires a certain amount of trial and 
error. Few program improvement teams 
will identify all of the most relevant and 
appropriate information on the first try. 
When team members sit down to exam- 
ine their indicator data, they will most 
likely discover additional data needs and 
may decide to drop or add some data 
sources and indicators. It is the nature of 
performance indicator systems to raise 
more questions than they answer. How'- 
ever, this is their greatest strength- 
generating discussion and debate based 
on objective, if imperfect, data rather 
than on hunches, anecdotal evidence, or 
the force of inertia. 

Establishing a Program Improve- 
ment Process 

Over time, the performance indicator 
system provides schools with trend data 
to determine whether improvement 
strategies appear to be working. Program 
improvement team members meet peri- 
odically to review the indicator data; 
determine whether performance is im- 
proving; discuss reasons why improve- 
ment is or is not happening; and refine 
their indicators and impiovement strate- 
gies. The team also decides with whom 
to share the performance information. 
Some teams may develop a school re- 
port card that is distributed periodically 
to students, par^'nts, faculty, district ad- 
ministrators, and the school board. 
Others may post student and teacher at- 
tendance rates, for example, on a daily 
basis in an attempt to generate some 
healthy competition. Still others may de- 
cide that classroom-specific performance 
data should be reviewed only by the 
school principal. Whichever specific dis- 
semination strategies are employed, 
participants in the program improve- 
ment process decide together how best 
to use the indicator information to bring 
about improved performance at their 
school. 






Conclusion 

Locally developed performance indi- 
cator systems offer a promising strategy 
for establishing a data-based program 
improvement process in districts and 
schools. By encouraging local educators 
to articulate their goals and involving 
them in deciding how to measure their 
performance on the goals, the model de- 
scribed here ensures that indicator 
systems will be relevant to local educa- 
tional objectives. After working through 
the process, educators should also be- 
come familiar w'ith the many data 
available to them and begin to see the 
data’s usefulness for answering a wide 
variety of questions about performance 
and effectiveness. Indicator systems are 
also generally practical and feasible, with 
all districts and schools having access to 
at least some meaningful data. In these 
ways, indicator systems avoid several 
pitfalls of other data collection and re- 
porting strategies. 

- Karen Levesque, Denise Bradby, 
and Kristi Rossi, MPK Associates, Inc. 

This CenterFocus was developed at the 
Institute on Education and the Econ- 
omy, Teachers College, Columbia 
University, which is a site of the Na- 
tional Center for Research in Vocational 
Education. 
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